THE ROLE OF EXPERT JUDGMENT IN 
RISK ANALYSIS 




1 . 


USE OF EXPERT JUDGMENT IN RISK ANALYSIS 

a. There is no Question That Expert Judgment is 
Needed; The Only Question is Whether to Use 
11 Explicit ly 

b. It Requires No More Use of "Art" Than Any 
Other Field 

c. Expert Judgment May Not Provide the Solution; 
However, It Will Often Show You Where and How 
to Seek It 

d. Properly Applied, Expert Judgment Can Assist 
In: 


Identifying Decision Alternatives; 

Validating Analytical Models; 

Assessing Influencing Parameters 

Note: False Assumptions Relative to 

A Parameter, Such as Its Degree of 
Independence, Can Significantly Affect 
The Final Results 

e. Expert Judgment Should Reflect Current Knowledge 
and Quantify Attendant Uncertainties; While the 
Additional Information Provided Will Help Reduce 
Ignorance, It Will Not Necessarily Reduce Uncertainty 

f. Although Expensive, Its Use "Up Front" Can be Very 
Cost-Effective in Terms of the Avoidance of Delays 
and Associated Costs at Later Stages 
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2. 


HOW MUCH DATA ARE SUFFICIENT 


a. Expert Judgment is not a Substitute for Data When 
It can be Readily Obtained 

b. Expert Judgment Should Follow Only After an Agreed 
Upon Level of Effort to Perform Additional Scientific 
Studies, and to Analyze and Evaluate the Resulting Data, 
Has Been Met 


3 . 


OBSERVATIONS ON THE USE OF EXPERT JUDGMENT 

a. It Is Well Established as a Tool in Risk Analysis 

b. Its Use is Mandatory Where Data are Inadequate; However, 
It Should Not Be Used as a Replacement for "Hard” Data 

Co Proper Application of Expert Judgment Requires Care, 

Planning, Documentation, and Resources. Protocols for 
Elicitation Should Be Systematic, Visible, and Easily 
Understood 


d. It Should Not Be Conducted as Simply a Poll of Experts; 

It Should Include Explicit Articulation of the Principles, 
Reasoning, and Data on Which the Judgments are Based 

e. Its Formal Solicitation Can and Should Clearly Explicate 
the Uncertainties for the Decisionmakers; At the Same 
Time, It Should be Recognized That Such Judgments 
Represent Only "Snapshots" in Time of Prevelant Opinion; 
They Are Not a Means for "Revealing" Truths 

f. The Methodology Used Can Significantly Influence the 
Result of the Process; Key Factors Include: 

Identification and Selection of Issues; 

Identification, Selection, and Use of Experts 
and Solicitors; 

The Biases of the Expert and Elicitors; 

The Methods for Aggregating the Results 

g. An Important Issue is Estimating the Magnitude of the 
Uncertainty, Especially Where Data Are Limited 

^ h. In All Cases, the Selected Approach (or Conclusion) 

Should be Justified; The Most Conservative Approach 
Should Not Necessarily Be Chosen Simply Because the 
Data Are Inadequate 
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4. 


OBSERVATIONS FROM PAST USE OF EXPERT PANELS 


a. There Is Always a Lack of Sufficient Time for 

Adequate Iteration With the Experts on Their Results 

b* The Definition of Issues is Always Critical; This 

Involves a Balance Between the Freedom of the Expert 
and Overdefining and Overconstrueting the Issues 

c. Anonymity Should be Avoided; Knowledge of the 
Sources of Specific Opinions Improves The 

Defensibi1ity of the Results 

d. Each Expert Should be Encouraged to Analyze the 
Questions Indepth Prior to the Panel Being Convened 

e ■» A Full Range of Weighting Strategies Should be 
Evaluated in Aggregating the Results 


5. QUESTIONS TO BE CONSIDERED IN THE USE OF EXPERT JUDGMENT 

a. How Should the Experts Be Identified and Selected? 

b* What Are The Preferred Methods for Using Formal 
Expert Judgment? How are the Mechanics of the 
Elicitations to be Carried Out? 

c. How Are the Pertinent Issues and Questions to be 
Selected? 

d. What Are The Pitfalls Associated With the Use of 
Expert Judgment? For Example, How Can Biases 
(Institutional and Motivational) Be Evaluated 
and Reduced? 

e. Is Dependency Among Experts (Mutual Influence) Desirable? 
If so, What Are the Appropriate Ways to Encourage It? 

If not. How Can It Be Minimized or Eliminated? 

f. Are There Appropriate Protocols for Weighting the 
Judgments of Experts and for Weighting Alternative 
Models and Their Results? If so, What Are the Criteria 
for Assigning Weighting Factors? 

g. How Should Judgments Be Aggregated? What Are The 
Criteria for Choosing an Aggregation Method? 

h. What is the Appropriate Level of Documentation for the 
Elicit ation? 

i. What Steps Should be Taken to Control the Potential 
Influence of Normative Experts on the Outcome? 
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6 . 


QUESTIONS ON ELICITATIONS FROM SINGLE EXPERTS 


a. Lack of Benefit of Interaction with Peers 

b. Assumptions Underlying- Judgments May Not Be 
Articulated 

c. Expert May be More Subject to Motivational 
and Cognitive Biases 

d. Quantification of Uncertainty Will Generally 
be Less 

e. Results May be Less Defensible; A Single Expert 
(Versus a Panel) May Have Less Credibility 
With Outside Groups 


7. SELECTION AND ORGANIZATION OF PANEL MEMBERS 

a. They Can Be Selected on the Basis of: 

Their Publications; 

Their Relevant Experience; 

Their Experimental Knowledge; 

Their Degree of Cooperation; and 
The Input From Outside Groups 
(These Can Be Solicited by Mail 
or Through an RFP) 

b. Should They Be Organized Horizontally of 
Vertically? 

Vertical Interaction is Helpful; 

Horizontal May Make Interfaces Difficult 
(They Must Interact to Exchange Information 
at Each "Nodal” Point) 


8 . 


TRAINING OF PANEL MEMBERS 

a. Is Required for Essentially All Panel Members 

b. Should Be Designed to Sensitize the Experts 
on How Their Information Will be Used and How 
It Could Be Abused 

c. Should Recognize That Motivational Biases (Hidden 
Agenda) Are Less Responsive Than Cognitive Biases 


4 


Source: https://www.industrydocuments.ucsf.edu/docs/lfclOOOO 


2025546022 



9♦ PITFALLS AND ADVANTAGES OF THE PANEL APPROACH 

a. Dominant Personalities May Heavily Influence 
the Results: 

Avoid by Having Each Member Formally 
Document His/Her Opinion and Rationale; 

Experts Can Also Be Required to Construct 
Probability Distributions for Their 
Positions 

The Greatest Sources of Bias Are Unstated 
Assumptions 

b. To Avoid Criticism, It is Often Necessary to 
Use "Experts" With Relatively Shal1ow-Knowledge 
of the Subject: 

This May Require Training That Can 
Introduce Biases 

c. To Overcome Biases, Panels Can be Made Up of 
Experts With Diverse Perspectives: 

While This Can Remove Some of the Biases, 

It May "Fix" the Results 


10 . 


BENEFITS OF THE PANEL APPROACH 


a. All Members Can Be Provided the Same Information 

b. Approach or Thoughts of Each Will Be Transparent 
for Other Members to Review 

c. Positions Will Change As a Result of New Information 
Provided and Group Dynamics 
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11. 


STEPS IN THE PANEL PROCESS 

a. The Issues Must be Identified and Selected 

b. The Experts Must be Identified and Selected 

c. The Issues Must be Discussed and Refined 

d. The Experts Must be Trained for Elicitation 

e. The Information Must be Elicited 

f. The Results Must be Analyzed and Aggregated, 
and Disagreements Resolved 

h. The Results Must Be Documented and Communicated 
to the Needed Parties 


12. GUIDANCE FOR GOOD PANEL OPERATIONS 


a. Ambiguity Can Be Minimized by Making the Questions 
Specific 

b. If Final Conclusions Are to be Aggregated, They 
Must Be in the Same Format 

c. If an Expert Has Difficulty Quantifying an Answer, 
Decompose the Question for Which He/She Can Construct 
Answers 
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13. 


GUIDANCE ON PANEL METHODOLOGY 

a. The Methodology Must Allow Different Experts to 
Share Each Others r Information so that Each Expert 
Can Base His/Her Ultimate Judgment on a Common Set 
of Information 

b. The Methodology Must Provide for Significant Inter¬ 
actions Among the Experts, So That They Can Share 
Different Perspectives and Approaches 

c* The Methodology Must Allow The Experts to Define or 
Redefine the Questions Being Asked, So That They 
Can Answer the Questions in Their Terras 

d. The Methodology Must Lead to the Conduct of the 
Proceedings in a Manner so as to Motivate the 
Experts 

e. The Methodology Must Provide Structured Ways in Which 
"Biases" (Different Perspectives) Can Be Identified, 
Brought to the Surface, and Subjected to Scrutiny 

f. The Methodology Must Allow Sufficient Time so That 

the Experts Can Perform Their Own Research and Analysis; 
Usually This Calls for Two Iterations in the Elicitation 

g. The Methodology Must Recognize That the Questions Being 
Addressed Do Not Have One "Correct" Answer, or Even "One" 
Correct Approach 

h. The Methodology Must Provide Full Documentation of What 
Happened, What Factors Underlie the Judgments of Each 
Expert, and Why; This Means That Each Expert Must be 
Comfortable With the Way in Which His/Her Judgment is 
Documented 
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AGGREGATION OF CONCLUSIONS OF INDIVIDUAL PANEL MEMBERS 


a. The Opinions of "Mainstream" Experts Can be 
Requested to Evaluate the Expertise of Individual 
Panel Members 

b. If a Rationale Exists to Reject the Opinion of One 
or More Panel Members, This Can Be Done; However, 

Any Quick Ploy to Remove Dissenting: Opinion Must 
Be Avoided 

c. Oftentimes It is Important to Retain the Component 
Contributions of Individual Panel Members, Even if 
Only One Aggregated Result is Needed in the Particular 
Application 

d. Forced Consensus Compromises the Defensibi1ity of the 
Results 


15 . 


DOCUMENTATION OF THE ELICITATION 

a. It Should Include Unambiguous Specification of * 
the Events for Which Probabi1ities Were Assessed 


b. The Assessed Probabi1ities for Those Events Should 
be Clearly Expressed 

c. The Data, Reasoning, Models, and Calculations Used 
in the Assessments Should be Documented, As Well As 
How and Why They Were Used 

d. The Documentation Should Also Include a Rational 
Basis for Eliminating and/or Not Documenting Any 
Information That Was Considered to be "Obvious" 
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CONCLUSIONS 


a. Expert Judgment is a Necessary Part of Risk Analysis 

b. Expert Judgment Must be Applied in Both an Explicit and 
Implicit Manner, With a Reasonable Division Between the 
Two 

c. The Final Product Should Include Detailed Information 
on the Principles, Rationale, and Data On Which the 
Conclusions are Based 

d* Where There are Major Weaknesses in the Experimental 
Data, the ’'Strengths” of Expert Judgment Can Be 
Extremely Helpful 

e* Explicit Expert Judgment is Primarily Designed to Aid 
in Decision Making; It is Not There^ Solely to Provide 
a "Warm Fuzzy Feeling” 1 


17. OTHER CONCLUSIONS 

a. Where the Reasoning Process Can Be Understood (and 
Accepted by Other Reasonable People), A Decision 
Based on Expert Judgment Should Stand Up in the 
Cour t s 


b. Although the Process of Solicitation of the Experts 

is Important, the Legal System Will Be More Interested 
in the Facts and Rationale for the Decision 

c. The Primary Purpose of Expert Judgment is to Provide 
a Foundation on Which the Decisionmakers Can Act 


d. Expert Judgment is Always Subject to Change and 
Should be Interpreted Accordingly 
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